[windows] add Python 3.10.1 to the installer #447

charles-zablit · 2025-08-01T18:10:22Z

This patch adds an option to install the embeddable version of Python directly from the toolchain installer on Windows.

The installer will pick up the python files from T:\Program Files\Swift\Python and install them in the bin directory alongside lldb.exe.

This is in continuation of the work in swiftlang/swift#83488.

rdar://157773554

mhegazy · 2025-08-01T19:36:58Z

platforms/Windows/python/python.wixproj

@@ -0,0 +1,5 @@
+<Project Sdk="WixToolset.Sdk/4.0.5">
+  <PropertyGroup>
+    <OutputName>python</OutputName>


since this is going be part of the toolchain, we will need to have one per variant. here is my last change for how to make the authoring work for all variants: #444

i can help make that happen if needed.

Do we really need 1 build per variant? I think it should be possible to share the Python installation across the different toolchain variants. We should lay them out such that they are isolated:

[INSTALLROOT]\Swift\Python-3.10.1\...

If we layout python this way, it would be isolated from any system installations and should be possible to share across different variants of the toolchain.

[INSTALLROOT]\Swift\Python-3.10.1\...

Just to make sure I understand, would the Embeddable Python actually live in [INSTALLROOT]\Swift\Python-3.10.1\...? As in, once installed, python.exe will be at [INSTALLROOT]\Swift\Python-3.10.1\python.exe?
In that case, how would lldb resolve the path to the embeddable python?

My initial plan was to extract the Python files into T:\Program Files\Swift\Python like you suggested here and then install them in the bin directory alongside lldb.exe.

The installation information should be queryable and you can extract that from the registry. This is basically what tcrun does as well.

mhegazy · 2025-08-01T19:39:05Z

platforms/Windows/python/python.wxs

+      Version="$(NonSemVerProductVersion)"
+      Scope="$(PackageScope)">
+
+    <Media Id="1" Cabinet="$(VariantCabinetName)" EmbedCab="$(ArePackageCabsEmbedded)" />


this is not going to build without these variables defined. i would recommend moving this file to a wxi, and defining these variables in a wxs file that imports it. see https://github.com/swiftlang/swift-installer-scripts/blob/main/platforms/Windows/bld/asserts/bld.asserts.wxs

Fixed, thanks! However, I think the file structure in swift-installer-scripts\platforms\Windows\python is not correct: the subdirectory should not be named asserts. Do you have a suggestion of what this should be named? Or should I just adopt a flat directory structure?

I think that cpy.wxs (as this is CPython) is fine as a name.

Right now, the file structure is:

python/ ├─ asserts/ │ ├─ python.wxs │ ├─ python.wixproject python.wxi

I will rename the python files to cpy but what should the asserts folder be named?

@mhegazy regarding the variant name, I would really like to remove asserts for Python as it's not an asserts build, just a regular Python distribution. Would the following file structure be OK?

python/ ├─ python.wxs ├─ python.wixproject ├─ python.wxi

this makes a lot of sense to me thanks!.

If lldb can look up an env var for LLDB_PYTHON_HOME we can set that during setup time to be the location where we install python

I have tried to install the embeddable Python alongside the Asserts and NoAsserts directories, with LLDB_PYTHON_HOME=../../Python and tried other CMake defines like LLDB_PYTHON_EXE_RELATIVE_PATH.

This does not work because lldb still tries to look for python310.dll in the Path.

I see 2 possible solutions going forward:

1

Use Windows APIs to manually add specify the location of python310.dll before starting lldb. Since we optionally install Embeddable Python, this adds some hidden logic, if the user uses regular Python instead, in which case the DLL will actually be in the PATH. For this reason, I'm not in favor of this solution.

2

Put the contents of the embeddable Python folder in usr/bin. We would have something like this:

Asserts/ ├─ usr/ │ ├─ bin/ │ │ ├─ lldb.exe │ │ ├─ python.exe │ │ ├─ python310.dll NoAsserts/ ├─ usr/ │ ├─ bin/ │ │ ├─ lldb.exe │ │ ├─ python.exe │ │ ├─ python310.dll

I have tested this scenario and it works fine.

Now your concern here was regarding this scenario:

Asserts/ ├─ usr/ │ ├─ bin/ │ │ ├─ lldb.exe │ │ ├─ python.exe │ │ ├─ python310.dll NoAsserts/ ├─ usr/ │ ├─ bin/ │ │ ├─ lldb.exe

In this scenario, the Asserts variant works fine. The NoAsserts variant should work fine as well because Asserts/usr/bin will be in the PATH. If it's not, the user will have to install regular Python manually. I think this is OK because they would have had to install it either way. To I don't think that the following is true.

so if we only install to one, the other variant is broken since lldb expects it to be in the path, and it will not be.

I think we should go with the 2nd solution, because it works with minimal changes to lldb's code.

I do not think option 2 is a good one. Installing the component per variant is not the best path as we discussed earlier.

Here is the thing, we install usr/bin in the PATH. so if we put python.exe in usr/bin users will find it if they just run python. this is not diffrent from putting it in ../../Python and then adding this to the PATH so that lldb find it.

I think I have another option. lldb needs 2 things, python310.dll since it was linked against it, and cannot load without it, and the rest of the embeded python installation to execute script, and find imports etc..

For the first, we would add a new component under LLDB ComponentGroup for python310.dll - this will be in the PATH since the whole folder is, but it is not an issue since this is the dll and not the python.exe.

For the second, we would install the new python.msi to the ..\..\Python as we discussed before, and set LLDB_PYTHON_HOME=../../Python as part of the installation.

@compnerd what do you think?

That would be problematic as the python.exe (interpreter) is then going to fail to find the DLL. I think that the best option here is to do a LoadLibraryW for python3.10.dll and in the failing case use SetDllDirectory to inject the additional library search path. I just want to verify that it does not change the DLL lookup order if we do that (as in would block the previous search order from being honoured).

mhegazy · 2025-08-01T19:40:36Z

platforms/Windows/python/python.wxs

+      </Component>
+
+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\python.exe" />


do we really need the executable? i thought that all lldb needed is the dll and the built in modules?

Yes, we should include thee executable. Ultimately, the issue with all this is that LLDB is meant to be usable as a scriptable debugger, including for post-mortem scenarios. In such a case, you want to be able to execute a script, which is going to require the interpreter.

do we really need the executable? i thought that all lldb needed is the dll and the built in modules?

On Windows, following this discussion, we want to bundle an executable Python with the toolchain installer.

mhegazy · 2025-08-01T19:44:07Z

platforms/Windows/platforms/windows/windows.wxs

@@ -9,6 +9,7 @@
      Scope="$(PackageScope)">

    <?define PlatformRoot = "$(ImageRoot)\Platforms\Windows.platform"?>
+    <?define PythonRoot = "$(ImageRoot)\python"?>


this should be dedined in python msi project and not here.

Fixed, thanks!

compnerd · 2025-08-03T21:42:35Z

platforms/Windows/bundle/installer.wixproj

@@ -19,6 +19,7 @@
    <ProjectReference Include="..\bld\asserts\bld.asserts.wixproj" BindName="bld.asserts" />
    <ProjectReference Include="..\cli\asserts\cli.asserts.wixproj" BindName="cli.asserts" />
    <ProjectReference Include="..\dbg\asserts\dbg.asserts.wixproj" BindName="dbg.asserts" />
+    <ProjectReference Include="..\python\python.wixproj" BindName="python" />


Lets shuffle this down after all ide MSI. I wonder if it makes sense to rename this to match the 3-letter naming.

I moved it down below ide. Should it be at the very bottom?

Regarding the 3-letter renaming, I'm not opposed to it, however I think it would be good to differentiate between embedded Python and regular Python if we ever decide to bundle/chain the full Python msi with the toolchain installer.

If we are sure we are never going to chain the full Python msi, then we could name this py3?

I am certainly curious about why the differentiation matters.

Ultimately, the difference between the embedded and full python is something that I think that we should somewhat blur. The actual difference is the lack of an installer (not a problem), a slightly smaller standard library (is that truly a concern?), and the missing pip which we can inject. So, difference-wise it is pretty small.

They also do not author MSIs, which means that the chaining would be more complicated than if they provided a MSM/MSI.

compnerd · 2025-08-03T21:43:39Z

platforms/Windows/bundle/theme.xml

-            <Checkbox Name="OptionsInstallAndroidSDKAMD64" X="210" Y="363" Width="-11" Height="17" TabStop="yes" FontId="3" EnableCondition="OptionsInstallAndroidPlatform">#(loc.Sdk_ProductName_Android_amd64)</Checkbox>
-            <Checkbox Name="OptionsInstallAndroidSDKARM" X="210" Y="381" Width="-11" Height="17" TabStop="yes" FontId="3" EnableCondition="OptionsInstallAndroidPlatform">#(loc.Sdk_ProductName_Android_armv7)</Checkbox>
-            <Checkbox Name="OptionsInstallAndroidSDKX86" X="210" Y="399" Width="-11" Height="17" TabStop="yes" FontId="3" EnableCondition="OptionsInstallAndroidPlatform">#(loc.Sdk_ProductName_Android_x86)</Checkbox>
+            <Checkbox Name="OptionsInstallEmbeddedPython" X="192" Y="165" Width="-11" Height="17" TabStop="yes" FontId="3">#(loc.EmbeddedPython_ProductName)</Checkbox>


I think that this should come at the very end as it is not really part of the toolchain, it is a third party dependency we are installing.

Fixed, thanks 👍

compnerd · 2025-08-03T21:44:07Z

platforms/Windows/platforms/windows/windows.wxs

@@ -9,6 +9,7 @@
      Scope="$(PackageScope)">

    <?define PlatformRoot = "$(ImageRoot)\Platforms\Windows.platform"?>
+    <?define PythonRoot = "$(ImageRoot)\python"?>


compnerd · 2025-08-03T21:44:19Z

platforms/Windows/python/python.wixproj

@@ -0,0 +1,5 @@
+<Project Sdk="WixToolset.Sdk/4.0.5">
+  <PropertyGroup>
+    <OutputName>python</OutputName>


Do we really need 1 build per variant? I think it should be possible to share the Python installation across the different toolchain variants. We should lay them out such that they are isolated:

[INSTALLROOT]\Swift\Python-3.10.1\...

If we layout python this way, it would be isolated from any system installations and should be possible to share across different variants of the toolchain.

compnerd · 2025-08-03T21:44:58Z

platforms/Windows/python/python.wxs

+      Name="$(VariantProductName)"
+      UpgradeCode="$(VariantUpgradeCode)"
+      Version="$(NonSemVerProductVersion)"
+      Scope="$(PackageScope)">


How do we version this component? Should we version it as per the Python version or the Swift toolchain version?

compnerd · 2025-08-03T21:46:44Z

platforms/Windows/python/python.wxs

+      </Component>
+
+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\python.exe" />


Yes, we should include thee executable. Ultimately, the issue with all this is that LLDB is meant to be usable as a scriptable debugger, including for post-mortem scenarios. In such a case, you want to be able to execute a script, which is going to require the interpreter.

compnerd · 2025-08-04T15:34:51Z

platforms/Windows/python/asserts/python.wxs

+  <?define PythonRoot = "$(ImageRoot)\Python"?>
+
+  <?include ../python.wxi ?>
+</Wix>


Missing newline

Fixed, thanks 👍

platforms/Windows/python/asserts/python.wxs

compnerd · 2025-08-04T15:35:44Z

platforms/Windows/python/python.wxi

+
+    <ComponentGroup Id="EmbeddedPython">
+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\libcrypto-1_1-arm64.dll" />


Is this file included on both ARM64 and AMD64?

It's not, I've added the ArchSuffix define to address this. Thanks!

compnerd · 2025-08-04T15:35:54Z

platforms/Windows/python/python.wxi

+      </Component>
+
+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\libssl-1_1-arm64.dll" />


It's not, I've added the ArchSuffix define to address this. Thanks!

compnerd · 2025-08-04T15:37:27Z

platforms/Windows/python/python.wxi

+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\vcruntime140.dll" />
+      </Component>
+
+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\vcruntime140_1.dll" />
+      </Component>


This brings up the question - do we want to bundle an additional copy of the VCRuntime? This is an interesting question because the toolchain itself also does depend on the runtime. There is a copy in the system. And now in python. The system one however will get security updates, this will not. Since part of the argument for this upgrade is "security" - we shouldn't be bundling them IMO.

Removed them 👍

compnerd · 2025-08-04T15:38:10Z

platforms/Windows/python/python.wxi

+    <WixVariable Id="SideBySidePackageUpgradeCode" Value="$(VariantUpgradeCode)" />
+    <FeatureGroupRef Id="SideBySideUpgradeStrategy" />
+
+    <ComponentGroup Id="EmbeddedPython">


Can the directory attribute be moved to the ComponentGroup instead of the Component? That would simplify the authoring here.

Fixed, thanks 👍

compnerd · 2025-08-04T15:38:36Z

platforms/Windows/python/python.wxi

+      </Component>
+    </ComponentGroup>
+
+    <Feature Id="EmbeddedPython" AllowAbsent="yes" Title="$(VariantProductName)">


I don't think that we should allow this component to be absent. We would install nothing then.

Fixed, thanks 👍

compnerd · 2025-08-04T15:39:19Z

platforms/Windows/python/python.wxi

+
+      <Component Directory="toolchain_$(VariantName)_usr_bin">
+        <File Source="$(PythonRoot)\LICENSE.txt" />
+      </Component>


Why is the LICENSE file in /usr/bin? Shouldn't that be under /usr/share/licenses?

I did not know about usr/share/licenses. Created the directory and added the license there 👍

platforms/Windows/SideBySideUpgradeStrategy.props

platforms/Windows/bundle/installer.wixproj

platforms/Windows/python/asserts/python.wixproj

platforms/Windows/shared/swift.en-us.wxl

mhegazy · 2025-08-05T16:14:31Z

platforms/Windows/python/asserts/python.wxs

+  <?define VariantCabinetName = python.asserts.cab?>
+  <?define ToolchainVersionedVariantDirectory = ToolchainVersionedAsserts ?>
+  <?define VariantEnvironmentComponentGUID = 30629e0c-b376-47bc-bedf-fefb7d4ca61d?>
+  <?if $(ProductArchitecture) = "arm64" ?>


this is not variant specific. i would recommend moving it to the wxi to avoid it being duplicated for diffrent variants.

Yeah, this seems like we should sink it into the wxi.

platforms/Windows/bundle/installer.wixproj

compnerd · 2025-08-08T18:01:06Z

platforms/Windows/python/asserts/python.wxs

+  <?define VariantName = asserts ?>
+  <?define VariantUpgradeCode = $(PythonUpgradeCode)?>
+  <?define VariantProductName = !(loc.EmbeddedPython_ProductName)?>
+  <?define VariantCabinetName = python.asserts.cab?>


Suggested change

<?define VariantCabinetName = python.asserts.cab?>

<?define VariantCabinetName = python.cab?>

compnerd · 2025-08-08T18:01:24Z

platforms/Windows/python/asserts/python.wxs

+  <?define VariantUpgradeCode = $(PythonUpgradeCode)?>
+  <?define VariantProductName = !(loc.EmbeddedPython_ProductName)?>
+  <?define VariantCabinetName = python.asserts.cab?>
+  <?define ToolchainVersionedVariantDirectory = ToolchainVersionedAsserts ?>


Do we need the ToolchainVersionedVariantDirectory define now?

compnerd · 2025-08-08T18:01:31Z

platforms/Windows/python/asserts/python.wxs

@@ -0,0 +1,14 @@
+<Wix xmlns="http://wixtoolset.org/schemas/v4/wxs">
+  <?define VariantName = asserts ?>


Do we need the VariantName now?

compnerd · 2025-08-08T18:01:53Z

platforms/Windows/python/asserts/python.wxs

+  <?define VariantCabinetName = python.asserts.cab?>
+  <?define ToolchainVersionedVariantDirectory = ToolchainVersionedAsserts ?>
+  <?define VariantEnvironmentComponentGUID = 30629e0c-b376-47bc-bedf-fefb7d4ca61d?>
+  <?if $(ProductArchitecture) = "arm64" ?>


Yeah, this seems like we should sink it into the wxi.

compnerd · 2025-08-08T18:03:11Z

platforms/Windows/python/python.wxi

+    <FeatureGroupRef Id="SideBySideUpgradeStrategy" />
+
+    <ComponentGroup Id="EmbeddedPython" Directory="toolchain_$(VariantName)_usr_bin">
+      <Component >


Suggested change

<Component >

<Component>

compnerd · 2025-08-08T18:03:43Z

platforms/Windows/python/python.wxi

+    </ComponentGroup>
+
+    <ComponentGroup Id="EmbeddedPythonLicense">
+      <Component Directory="toolchain_$(VariantName)_usr_share_licenses">


I don't think that we should be adding this to the toolchain layout but rather the python subdirectory.

compnerd · 2025-08-08T18:04:01Z

platforms/Windows/shared/shared.wxs

@@ -55,6 +55,7 @@
                <Directory Id="toolchain_asserts_usr_share_docc" Name="docc">
                  <Directory Id="toolchain_asserts_usr_share_docc_render" Name="render" />
                </Directory>
+                <Directory Id="toolchain_asserts_usr_share_licenses" Name="licenses" />


This should move into the directory hierarchy for Python.

compnerd · 2025-08-08T18:05:35Z

platforms/Windows/readme.md

@@ -165,10 +167,11 @@ msbuild %SourceRoot%\swift-installer-scripts\platforms\Windows\bundle\installer.
  -p:Configuration=Release ^
  -p:BaseOutputPath=%PackageRoot%\online\ ^
  -p:ImageRoot=%ImageRoot%\Program Files\Swift ^
+  -p:PythonRoot=%ImageRoot%\Program Files\Python-3.10.1 ^


I think that we want to sink this further 1 level as %ImageRoot%\Program Files\Swift\Python-3.10.1. When we do that, we can remove the parameter as we only need the python version as we know the image root is %ImageRoot%\Program Files\Swift so we can compute the path relative to that as $(ImageRoot)\Python-$(PythonVersion).

compnerd · 2025-08-08T18:05:56Z

platforms/Windows/readme.md

  -p:Platforms="android;windows" ^
  -p:AndroidArchitectures="aarch64;armv7;i686;x86_64" ^
  -p:WindowsArchitectures="aarch64;i686;x86_64" ^
-  -p:WindowsRuntimeARM64=%ImageRoot%\Prograam Files (Arm64)\Swift\Runtimes\0.0.0 ^
+  -p:WindowsRuntimeARM64=%ImageRoot%\Program Files (Arm64)\Swift\Runtimes\0.0.0 ^


Hah! Thanks for fixing my typo.

charles-zablit requested review from shahmishal, compnerd and etcwilde as code owners August 1, 2025 18:10

charles-zablit mentioned this pull request Aug 1, 2025

[windows] upgrade to Python 3.10.1 embeddable swiftlang/swift#83488

Open

mhegazy reviewed Aug 1, 2025

View reviewed changes

compnerd reviewed Aug 3, 2025

View reviewed changes

[windows] add Python 3.10.1 to the installer

91159fc

charles-zablit force-pushed the charles-zablit/windows/embed-python-in-the-installer branch from d806397 to 91159fc Compare August 4, 2025 14:38

charles-zablit changed the title ~~[Draft][windows] add Python 3.10.1 to the installer~~ [windows] add Python 3.10.1 to the installer Aug 4, 2025

compnerd reviewed Aug 4, 2025

View reviewed changes

charles-zablit self-assigned this Aug 4, 2025

charles-zablit added 2 commits August 5, 2025 15:00

switch to using the PythonRoot parameter

888fd5f

address comments

796d69c

mhegazy reviewed Aug 5, 2025

View reviewed changes

platforms/Windows/SideBySideUpgradeStrategy.props Show resolved Hide resolved

mhegazy reviewed Aug 5, 2025

View reviewed changes

platforms/Windows/bundle/installer.wixproj Show resolved Hide resolved

mhegazy reviewed Aug 5, 2025

View reviewed changes

platforms/Windows/python/asserts/python.wixproj Show resolved Hide resolved

mhegazy reviewed Aug 5, 2025

View reviewed changes

platforms/Windows/shared/swift.en-us.wxl Show resolved Hide resolved

mhegazy reviewed Aug 5, 2025

View reviewed changes

compnerd reviewed Aug 8, 2025

View reviewed changes

	<?define VariantCabinetName = python.asserts.cab?>
	<?define VariantCabinetName = python.cab?>

		@@ -0,0 +1,14 @@
		<Wix xmlns="http://wixtoolset.org/schemas/v4/wxs">
		<?define VariantName = asserts ?>

[windows] add Python 3.10.1 to the installer #447

Are you sure you want to change the base?

[windows] add Python 3.10.1 to the installer #447

Uh oh!

Conversation

charles-zablit commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charles-zablit Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charles-zablit Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

1

2

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

charles-zablit commented Aug 1, 2025 •

edited

Loading

charles-zablit Aug 4, 2025 •

edited

Loading

charles-zablit Aug 4, 2025 •

edited

Loading